(Not Applicable)
1. Field of the Invention
This invention relates to optical metrology, and particularly to the problem of making accurate non-contact dimensional measurements of objects that are viewed through an endoscope.
2. Description of Related Art
2a. Perspective Dimensional Measurements with Endoscopes
In the past several decades, the use of optical endoscopes has become common for the visual inspection of inaccessible objects, such as the internal organs of the human body or the internal parts of machinery. These visual inspections are performed in order to assess the need for surgery or equipment tear down and repair; thus the results of the inspections are accorded a great deal of importance. Accordingly, there has been much effort to improve the art in the field of endoscopes.
Endoscopes are long and narrow optical systems, typically circular in cross-section, which can be inserted through a small opening in an enclosure to give a view of the interior. They almost always include a source of illumination that is conducted along the interior of the scope from the outside (proximal) end to the inside (distal) end, so that the interior of a chamber can be viewed even if it contains no illumination. Endoscopes come in two basic types; these are the flexible endoscopes (fiberscopes and videoscopes) and the rigid borescopes. Flexible scopes are more versatile, but borescopes can provide higher image quality, are less expensive, are easier to manipulate, and are thus generally preferred in those applications for which they are suited.
While endoscopes (both flexible and rigid) can give the user a relatively clear view of an inaccessible region, there is no inherent ability for the user to make a quantitative measurement of the size of the objects he or she is viewing. There are many applications for which the size of an object, such as a tumor in a human body, or a crack in a machine part, is a critically important piece of information. Making a truly accurate measurement under these circumstances is a long-standing problem that has not been adequately solved until recently.
In a first co-pending application, now U.S. Pat. No. 6,009,189, entitled xe2x80x9cApparatus And Method For Making Accurate Three-Dimensional Size Measurements Of Inaccessible Objectsxe2x80x9d, filed Aug. 16, 1996, and which is incorporated herein by reference, I taught a new and complete system for making measurements of objects with an imaging optical system, with particular emphasis on endoscopic applications. In a second co-pending application, now U.S. Pat. No. 6,121,999, entitled xe2x80x9cEliminating Routine Alignment Calibrations in Perspective Dimensional Measurementsxe2x80x9d, filed Jun. 9, 1997, I taught certain improvements to the measurement system as applied to the endoscopic application. I will hereinafter refer to the first co-pending application as xe2x80x9cApplication 1xe2x80x9d and the second as xe2x80x9cApplication 2xe2x80x9d.
My previous invention makes possible a new class of endoscopic measurement instruments of unprecedented measurement accuracy. This measurement system is a version of a technique I call xe2x80x9cperspective dimensional measurementxe2x80x9d. By xe2x80x9cperspectivexe2x80x9d I am referring to the use of two or more views of an object, obtained from different viewing positions, for dimensional measurement of the object. By xe2x80x9cdimensional measurementxe2x80x9d, I mean the determination of the true three-dimensional (height, width, and depth) distance(s) between two or more selected points on the object.
As a necessary and integral part of my complete measurement system, I taught how to calibrate it in the referenced applications. I taught the use of a complete set of robust calibration procedures, which removes the need for the measurement system to be built accurately to a specific geometry, and also removes any need for the imaging optical system(s) to be built accurately to specific optical characteristics. Instead, I taught how to calibrate the geometry and characteristics of the opto-mechanical hardware, and how to take that actual geometry into account in the measurement process. The complete set of calibration procedures I taught includes three different types of calibration. In optical calibration, the detailed characteristics of each imaging optical system (i.e., camera), when used as a precision image forming device, are determined. In alignment calibration, the orientations of each camera""s measurement coordinate axes with respect to the motion of the camera are determined. Finally, in motion calibration, any errors in the actual motion of the camera(s), as compared to the ideal motion, are determined.
In Application 2, improvements to the system were made that eliminated the necessity of repeating the alignment calibration in certain important circumstances.
In some embodiments, my previous invention enables one to make accurate measurements using a standard, substantially side-looking, rigid borescope. Since the person who needs the measurement will often already own such a borescope, the new method offers a significant cost advantage over earlier measurement techniques.
In other embodiments, my previous invention provides for new types of self-contained endoscopic measurement instruments, both rigid and flexible, which offer significantly improved measurement accuracy as compared with those previously available. I call these new instruments the electronic measurement borescope and the electronic measurement endoscope.
While my system, as previously disclosed, does produce accurate dimensional measurements, there is room for improvement. The problem is that a new optical calibration may have to be performed each time the focus of the instrument is adjusted. One of the parameters determined during optical calibration is proportional to the magnification of the image. Without making special provisions for it, the magnification will most likely not be constant with focus, and thus every time the instrument is refocused, there is the logical requirement for a new optical calibration. Additional parameters that are determined during optical calibration are the location of the optical axis on the image sensor and the distortion of the image. These parameters may also vary as the focal state (that is, the object plane that is in focus) of the instrument is changed, which would be additional reasons to require a new calibration. Of course, whether a new calibration would actually be required in any specific instance depends on the accuracy required of the dimensional measurement, and on the characteristics of the camera being used in that instance.
When a standard borescope is used with my previous invention, there is the further difficulty that when an image sensor is mounted to the borescope to perform perspective measurements and the assembly is then calibrated, this calibration is lost if the image sensor is subsequently removed from the borescope. One may wish to remove the image sensor temporarily either to use the borescope for visual inspection, or to use the image sensor with another borescope that has different characteristics. What is needed here is a way to allow the removal and replacement of the measurement image sensor while maintaining calibration of the measurement system.
2b. Magnification and Focus in Optical Metrology and Machine Vision
It is known that the magnification of an image formed by an optical system depends on the range of the object; that is, the magnification depends on the distance between the object and the optical system. It is also known that there is a well defined relationship between the position of a focusing component in an optical system and the range of an object that is in focus. These known relationships have been used in a class of endoscopic measurement instruments that implement a technique that I call measurement by focus. U.S. Pat. No. 4,078,864 to Howell (1978) and U.S. Pat. No. 5,573,492 to Dianna and Costello (1996) are examples of this approach. In these instruments, a focusing component is instrumented to produce a datum that is a function of the range to the object plane that is in focus. (By xe2x80x9cinstrumentedxe2x80x9d, I simply mean that the position of the focusing component along the system""s optical axis is measured with respect to some fixed reference within the system.) This datum is then used together with a calibrated relationship between the range and the magnification to produce a relationship between dimensions as measured on the image and the corresponding dimensions on the object.
The fundamental problem with this approach to making accurate dimensional measurements is that the position at which the image is in focus is difficult to determine, thus the range measurement, and hence the magnification, is subject to relatively large random errors (that is, a lack of repeatability). As a result, these instruments, as disclosed and in practice, are restricted to making two dimensional measurements, i.e., measurements of distances that are oriented perpendicular to the optical axis.
The key assumption of the measurement by focus technique is that there is a fixed relationship between the range of an object and its magnification, because it is assumed that an object will always be viewed in focus. This assumption cannot apply to any three-dimensional measurement technique because the measurement determines the depth of an object, while only a single plane of the object can truly be in focus at a time. In general the magnification of an image depends not only on the range of the object, but also on the range at which the optical system is focused, that is, on the focal state of the optical system.
The telecentric principle has often been applied to standard two dimensional optical measurements, such as those made by optical comparators. Telecentricity refers to the situation where the cone of light forming each point in an image has a central axis which is parallel to the optical axis of the system. It is only recently that telecentricity has been applied to three-dimensional machine vision applications, in a paper by M. Watanabe and S. K. Nayar: xe2x80x9cTelecentric Optics for Computational Visionxe2x80x9d, Lecture Notes in Computer Science, 1065, 1996. The applications that these authors consider are called xe2x80x9cdepth from focusxe2x80x9d and xe2x80x9cdepth from defocusxe2x80x9d and are closely related to the measurement by focus technique.
In depth from focus, the focal state of the imaging camera is varied in small steps throughout a range of focal states. An image of the scene of interest is acquired at each of these focal states. The images are then analyzed to determine in which image the individual elements of the scene are in best focus, thus obtaining a relatively crude estimate of the range of each element of the image. In depth from defocus, only two images are acquired at two focal states, and a different image processing scheme is used in an attempt to obtain the same information. As stated, these are machine vision applications, where the goal is a relatively coarse determination of the three - dimensional layout of the objects in a scene, rather than metrology, where the goal is a precise measurement of individual dimensions on an object. These techniques have the same problem as does the measurement by focus technique when it comes to accurate metrology.
Watanabe and Nayar point out that when a telecentric optical system is used and the image viewing plane is moved to focus the system, the magnification of the image does not vary with the focal state of the system. While they refer to this situation as xe2x80x9cconstant magnificationxe2x80x9d, I prefer to call it xe2x80x9cconstant relative magnificationxe2x80x9d. The first reason for defining a new term is that this is not the only type of constant magnification; there are other types of constant magnification which are important. Significant additional reasons for the use of my terminology will become apparent in the ensuing discussion.
While the system taught by Watanabe and Nayar meets the goals they set for it, their teachings are far from complete. As I will show, there are significant benefits to be gained from going beyond the use of a telecentric optical system for optical metrology. In addition, their teachings are in error in an important point, in that they state that their system will also work if the lens is moved with respect to the image viewing plane. I will show that this is incorrect in practice, and that the difference is important to accurate metrology.
In U.S. Pat. No. 4,083,057 (1978), Quinn disclosed the addition of a magnification corrector to an auto-focus lens system to correct for the change in magnification with focal state in video systems. The problem addressed by Quinn was that objects near the edge of the field of view of a video or movie camera would be seen to move into or out of the field of view as the focus of the camera was changed. At first glance, this problem is not related to machine vision or metrology, but Quinn""s system is the earliest example known to me of a non-telecentric system that may be able to achieve imaging at constant relative magnification. I say xe2x80x9cmay be able toxe2x80x9d because Quinn does not teach everything that is required to generate a correction which will work for objects at all ranges. I am not aware of any later work which remedies this deficiency.
A system that enables one to make photographs at constant magnification was disclosed by Yasukuni, et. al., U.S. Pat. No. 4,193,667 (1980), and has been followed by many other patents directed to the same end. In these systems, the goal is to image an object at a constant image size as that object is moved to various ranges, while also keeping the image in focus as it moves. I refer to this goal as imaging at constant absolute magnification. To perform this function, these systems use a variable focal length optical system combined with a focus adjusting component. Thus, these systems are adaptations of zoom or varifocal lenses.
A system for photocopiers, microform readers, and the like, which allows one to accurately set a variable magnification of the image of a fixed object, while simultaneously keeping the image in focus, was disclosed by Sugiura, et. al. in U.S. Pat. No. 4,751,376 (1988). This also has been followed by many other patents directed to the same end. These systems are also directed toward achieving a constant, and well-determined, absolute magnification. In this case, the focal length of the optical system is fixed, and the optical system is moved with respect to the object in order to change the magnification.
In all of the known systems providing constant absolute magnification, and in the known non-telecentric systems which may provide constant relative magnification, there are used two independently moving components in the optical system. These moving components are a magnification adjuster and a focus adjuster. The magnification adjuster is typically a lens group, while the focus adjuster is either a second lens group or an optical path length adjusting component. The relative motions of these components with respect to the object and with respect to each other are then controlled in a manner to produce the desired magnification and focus result. In the early systems described by Yasukuni, et. al. and by Quinn, the relative motions are controlled by mechanical cams. In later devices, motors are used to move the components, and position transducers are used to monitor the positions of the components.
For the purposes of endoscopic dimensional measurements, the fixed focal length constant absolute magnification systems are not applicable, since the distance between the object and the optical system cannot, in general, be controlled to adjust the magnification. The variable focal length systems could be applied to these measurements as well as to general optical metrology, but systems using two instrumented moving components are complex and expensive. In addition, because the disclosed systems have been designed for photography and not metrology, they do not produce the information necessary for accurate metrology.
In the general art of optical metrology, little attention has been paid to the detailed characteristics of the magnification in out of focus images, other than the use of the telecentric principle. It has not been well understood that accurate measurements can be made with out of focus images. The only related reference that I am aware of is an early letter by W. Wallin, xe2x80x9cA Note on Apparent Magnification in Out-of-Focus Imagesxe2x80x9d, Journal of the Optical Society of America, 43, 60, 1953. In this paper Wallin states that the magnification is defined only for the image plane, but then goes on to define what he calls an xe2x80x9capparent magnificationxe2x80x9d for out of focus images. Wallin""s comment is clearly incorrect, as any real image will almost never be in perfect focus, yet such images can be used for metrology. Wallin gives expressions for the apparent magnification of an optical system in a fairly general, obscure, and unusable form. He states that these expressions are useful for tolerancing optical comparators. What this paper lacks is any information or insight into how to design or improve the design of a metrological optical system using his concept of apparent magnification.
In the general art of perspective dimensional measurements, typically the object to be inspected is brought to an inspection station that has a set of fixed viewing cameras. Thus, the range of the object is essentially fixed, and so are the focal states of the measurement cameras. If the cameras are refocused for some reason, then the system is recalibrated. Although this system is satisfactory for many purposes, I believe that if the capability to refocus the cameras on an object of interest without also requiring a recalibration were available, it would be found useful in some of these standard inspection setups.
The most important alternative to perspective dimensional measurements in three-dimensional optical metrology is known as use of xe2x80x9cstructured lightxe2x80x9d. There are, for instance, commercially available endoscopic measurement systems based on this principle. These systems have the same problem with requiring recalibration if the camera is refocused, and would also benefit if this problem were solved.
Fundamentally, what is needed for the purposes of three-dimensional optical metrology, including perspective dimensional measurements, is a way to determine the magnification that applies at each point in an image, regardless of whether that point happens to be in focus or not; and this magnification must be determined as the focal state of the optical system changes. In addition, and especially in endoscopic applications, one must also determine any deviation of the optical axis and any change in the distortion in the image as the focal state is changed. This problem has heretofore not been addressed in a comprehensive or coherent manner.
In order to meet these goals with a fixed focal length optical system, one must determine how the magnification varies with two independent variables: the range of the object point and the focal state of the optical system. With a variable focal length system, there are three independent variables: the range, the focal state, and the focal length of the system. None of the prior art systems is capable of providing the required information about how the magnification, the position of the optical axis, and the distortion depend on the independent variables, nor can they incorporate any such information into the measurement. Thus, the teachings of the prior art are not sufficient to enable one to perform accurate three-dimensional metrology while also allowing an adjustment of focus to best view an object of interest, without also requiring a recalibration when the focus is adjusted.
Accordingly, the present invention has two overarching goals, which are:
1. Provide apparatus and methods that enable a user to adjust the focus of an optical metrology system while maintaining measurement accuracy, without requiring recalibration.
2. Make the apparatus as simple and inexpensive as possible.
In order to logically and systematically gain control of the independent variables in an optical system so that these goals can be met, I have developed several new concepts related to an optical system. Using these concepts, I have been able to reduce the number of separately instrumented moving components that are necessary in a variable focus metrological optical system. I have also been able to reduce the requirements on the precision to which the focus motion must be instrumented.
As previously stated, what must be determined is the relationship between the range of an object and the magnification of that object. This relationship must be determined for every possible focal state of the optical system. I have found that an optical metrology system can be simplified if the required relationship can be expressed by the following equation:
Magnification=(focus dependent part)xc3x97(range dependent part)xe2x80x83xe2x80x83(1)
When the magnification of an optical system can be expressed by Equation (1), then I say that the optical system exhibits a relative magnification, where the relative magnification is the focus dependent part.
In an optical system that exhibits a relative magnification, the range dependent part of the magnification is a fixed function of the range, and it is only the focus dependent part that must be considered when determining how the magnification of an object varies with focal state. When the focal state is changed, the magnification of an image changes by a factor that is the same for objects at any range, and thus is independent of the value of the magnification. In this situation, if the system is appropriately designed, only a single quantity must be instrumented to determine the information required to make accurate dimensional measurements. I have found that there are a large class of optical systems that exhibit a relative magnification, and I teach the requirements for this condition.
If, in addition, the focus dependent part of the magnification is a constant, then the magnification of the image does not depend on the focal state at all, and I refer to that situation as imaging at constant relative magnification. In this condition the image of an object at a given distance from a camera remains at the same size as the focus of the camera is varied, or, to state it another way, the image of a given object remains the same size whether it happens to be in focus or not. The size of the image will vary with the range of the object, but that size is independent of what range happens to be in focus. When an optical system exhibits a constant relative magnification, there is a fixed relationship between range and magnification, and that fixed relationship does not depend on the image being in focus. In this case, with proper system design, nothing at all has to be instrumented in order to obtain the required information for metrology.
The only previously known ways to achieve constant relative magnification were either to use an appropriate two moving component scheme to adjust both focus and magnification or to use a certain type of telecentric optical system. Not every two moving component system or every telecentric system will provide constant relative magnification. I teach the requirements which must be met by these types of systems in order to achieve that condition. I also teach the specific requirements that must be met by any optical system in order to achieve constant relative magnification. As a result of this new understanding, I have discovered simple, single moving component, non-telecentric optical systems that exhibit constant relative magnification, and which are both practical and useful for optical metrology.
If the relative magnification of an optical metrology system is not constant with focal state, then one must accurately determine any changes in the relative magnification, and one must also take those changes into account in the dimensional measurement. In this case there is no fixed relationship between range and magnification. I have developed apparatus and methods that make such systems practical and useful.
When an optical system does not exhibit a relative magnification, then the change in magnification when the focal state is shifted depends on the range of the object. In other words, the change in magnification for a given change in focal state depends on the magnification. I have found that there exist simple systems for which only one quantity must be instrumented in order to handle even this case, but the effort necessary to calibrate these systems is larger than that required for systems that do exhibit a relative magnification.
Therefore, several objects and advantages of the present invention are:
(a) to provide apparatus and methods that enable accurate perspective dimensional measurements to be made with an endoscope, while allowing a user to focus on an object of interest, without requiring more than an occasional optical calibration of the system;
(b) to provide apparatus and methods that allow a single measurement accessory camera to be used with any one of several endoscopes, and to be interchanged between them, while also providing accurate measurements, without requiring more than occasional recalibrations;
(c) to provide apparatus and methods that enable accurate dimensional measurements to be made with an electronic measurement borescope, or with an electronic measurement endoscope, while allowing a user to focus on an object of interest, without requiring more than an occasional optical calibration of the system;
(d) to provide optical systems for metrology cameras that enable accurate dimensional measurements to be made while also allowing a user to focus such cameras on an object of interest, without requiring more than an occasional optical calibration of the cameras;
(e) to provide optical systems for metrology that enable one make accurate measurements, while also allowing one to focus on an object of interest, without requiring the instrumentation of both the focal state and the focal length of the optical systems;
(f) to provide non-telecentric optical systems for optical metrology that enable one to focus on an object of interest, while maintaining measurement accuracy, without requiring the instrumentation of either the focal state or the focal length of the optical systems; and
(g) to provide the requirements for and a general method of designing non-telecentric optical systems that exhibit constant relative magnification.
In a first embodiment, my system provides an accessory measurement camera for an endoscope that incorporates a telecentric optical system. The focal state is varied by moving an image sensor along the optical axis, where the motion is preferably constrained to an accurate straight line by a linear translation stage. I teach how the optical system of the accessory camera can be designed to correct for aberrations of the endoscope in a manner to produce more accurate measurements.
My system also provides for a measurement mechanical interface and for a sufficient set of alignment adjustments to be incorporated into the interface so that a measurement accessory can be mounted or re-mounted to a endoscope and so that the measurement accessory can also be interchanged between endoscopes without requiring any recalibration of the measurement system.
I teach how to determine the relative magnification and deviation of the optical axis between two calibration images, and further, how to use this method to align measurement systems according to this invention so that the relative magnification and optical axis position are constant with focal state. I also teach how to determine the number of image points that must be contained in the calibration images to obtain system alignment to the required degree of precision for the desired metrological performance.
In another embodiment, my system provides an accessory focusing measurement adapter that adapts a video camera back to an endoscope. This adapter uses a single lens group that moves along the optical axis to adjust the focal state. The position of the lens group is instrumented with a position transducer. I teach how to optimize the layout of this system to minimize the change in relative magnification with focal state over the desired range of object distances, thus allowing one to obtain accurate metrology with an inexpensive position transducer.
I teach how to determine the relative magnification and the position of the optical axis of an optical system as a function of focal state. The method includes use of a single set of calibration images, as well as the combination of the results from multiple sets of calibration images obtained with calibration targets at different ranges. I further teach how to use these methods to compensate for the errors of embodiments of the present invention that are imperfectly aligned and/or that do not exhibit perfectly constant relative magnification. I also teach how to incorporate these relative magnification and optical axis location data into the perspective dimensional measurement.
In another embodiment, my system provides an accessory camera or focusing adapter to be used with an endoscope in which the optical system is telecentric, but in which a lens group, rather than an image sensor, is moved along the optical axis to change the focal state. Optionally the moving lens group can be designed to correct for the aberrations of the endoscope.
In other embodiment, my system provides an optical system for optical metrology that includes an aperture stop and a lens, considered as a single lens group, which are disposed at a fixed relative distance. The system also includes an image sensing plane which is disposed at a variable distance with respect to the lens and stop, and a position transducer which instruments the relative positions of the image sensing plane and the lens.
In another embodiment, my system provides an optical system for optical metrology that includes a unit comprising an aperture stop and a lens, considered as a single lens group, that are disposed at a fixed relative distance. The stopxe2x80x94lens unit is disposed at a variable distance with respect to an image sensing plane in order to vary the focal state of the system. The relative positions of the image sensing plane and the stopxe2x80x94lens unit are instrumented by a position transducer.
In another embodiment, my system provides an optical system for optical metrology in which an aperture stop and an image sensing plane are disposed at a fixed relative distance, and in which a lens, considered as a single lens group, is disposed at a variable position along the optical axis with respect to the stop and image sensing plane, and in which the relative positions of the lens and the stopxe2x80x94sensing plane unit are instrumented with a position transducer.
In another embodiment, my system provides a two lens group optical system that exhibits constant relative magnification, but is not telecentric. A second lens group and an image sensing plane are disposed at a fixed relative distance, and this unit is disposed at a variable distance with respect to a first lens group. The first lens group is also disposed at a particular location with respect to an aperture stop. As an option, the relative position of the second lens groupxe2x80x94image sensing plane unit with respect to the first lens group can be instrumented in order to allow for correction of imperfect alignment or of aberrations.
In another embodiment, my system provides a three lens group optical system that exhibits constant relative magnification, is not telecentric, but that maintains a near-telecentric condition as the focal state is varied. In this embodiment, a stop, first and third lens groups, and an image sensing plane are disposed at fixed relative positions along an optical axis. The first lens group is disposed at a particular distance with respect to the stop, and the third lens group is disposed at a particular distance with respect to the image sensing plane. A second lens group is disposed between the first and third lens groups, and its position along the axis is varied in order to change the focal state of the system. As an option, the relative position of the second lens group with respect to the rest of the system can be instrumented in order to allow for correction of imperfect alignment or of aberrations.
Further objects, advantages, and features of my system will become apparent from a consideration of the following description and the accompanying schematic drawings.